As the processing power available to devices and associated support services continues to increase, it has become practical to interact with users in new ways. In particular, it is becoming practical to interact with users through speech. Many devices are now capable of receiving and responding to voice commands, including personal computers, smartphones, tablet devices, media devices, entertainment devices, industrial systems, voice-based assistants, and so forth. In some instances, a user of such devices may include multiple devices in a single environment, where each device can each capture the speech from the user and attempt to interact with the user in response.